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Background : 

The  more  Intensive  application  of  the  concepts  and  techniques  of 
reliability  growth  management  was  one  of  the  highlights  of  a study  by 
the  Panel  on  "Accelerated  Development  of  Reliability"  which  was  presented 
to  the  Commanding  General  of  the  Army  Materiel  Command  in  February  of 
1972.  The  "Hope  Committee  Report"  as  it  became  known,  resulted  in  a much 
accelerated  development  and  application  of  the  reliability  growth  techniques. 
The  Army  Materiel  Systems  Analysis  Agency  (AMSAA)  assumed  a leading  role 
in  the  development  of  these  techniques  by  sponsoring  two  reliability  growth 
s>’mposia.  The  first  was  held  September  26  and  27,  1972  at  Aberdeen  Proving 
Grounds.  The  second  Ad-’SAA  reliability  growth  symposium  was  held  November  12 
and  13,  1974,  again  at  .Aberdeen  Proving  Grounds.  The  first  symposium 
emphasized  the  fundamental  conceptual  and  mathematical  aspects  of  re- 
liability growth  tracking  and  prediction  techniques,  whereas  the  second 
symposium  presented  fite  excellent  examples  of  the  application  of  the 
reliability  growth  manasement  methods  by  Project  Management,  Contractor,  and 
tt  various  commands.  Much  credit  for  the  excellent  progress  made  has  to  go 
to  Mr.  Pat  Bruno  for  the  vision  and  encouragement  given  as  well  as  Dr.  Larry 
Crow  for  the  technital  innovation  and  leadership. 

This  latest  study  crew  out  of  several  discussions  between  Mr.  S.J.  Lorber 
and  myself  as  wa  noted  the  increasing  acceptance  of  the  reliability  growth 
management  techniques  and  our  joint  concern  about  the  proper  balance  between 
technical  sophistication  and  the  practicalities  of  everyday  application. 

On  April  2,  1975,  Mr.  Lorber, representing  the  Deputy  Commanding  General, 
requested  that  I perform  a review  of  AMC's  use  of  reliability  growth  manage- 
ment in  my  capacity  as  a member  of  the  Army  Scientific  Panel.  The  charge 
i presented  to  me  by  Mr.  Lorber  was: 
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a)  Is  the  reliability  growth  effort,  as  currently  applied,  a 
worthwhile  activity  in  terms  of  aiding  the  decision-making 
processes  and  affecting  the  course  of  development  programs? 

b)  Specifically,  has  the  reliability  growth  methodology  affected 
decisions  at  the  technical  and  managerial  level? 

c)  How  can  the  reliability  growth  concept  application  be  strengthened 
and  improved? 

Because  of  other  comnitments,  I only  started  the  review  at  the  end 
of  1975  and  conducted  two  visits  in  1976  as  follows: 

1)  Initiation  visit  to  AMC  Headquarters  27  January  1976  to  meet 
with  S.J.  Lorber,  Art  Nordstrom,  Dan  Kruvand,  Jack  Lavery,  and 
Colonel  Donovan,  DSCRDA. 

2)  Visit  to  AMSAA  26  April  1976  to  meet  with  Pat  Bruno,  Larry  Crow, 
and  Jack  Lavery,  with  Colonel  Donovan  acting  as  Staff  Assistant. 

Much  information  was  also  obtained  at  TACOM  and  AVSCOM  in  connection  with 
ny  activities  as  Chairman  of  the  Product  Improvement  II  Ad  Hoc  Group. 

In  addition,  the  Proceedings  of  the  two  Reliability  Growth  Symposia, 
-articularly  the  second  symposium,  provided  much  needed  information. 

Lastly,  long  telenncne  calls  were  made  to  BG.  W.  Hllsman,  Project 
Manager  of  TACFIP.Z  one  ISQ  73,  MG  Lauer,  Project  Manager  of  UTTAS, 

Mr.  R.  Whitley,  forcer  Project  Manager  of  Dragon,  Mr.  Salter  formerly 
with  the  M60  Program  and  at  present  working  with  the  Project  Manager  of 
MICT,  and  Colonel  R.  Phillipp,  Project  Manager  of  XM198  project.  These 
candid  conversations  provided  an  up-to-date  look  at  the  present  picture  of 
reliability  growth  management  acceptance,  problems,  and  potential  for 


Improvement . 


SOME  REMARKS  ON  RELIABILITY  GROWTH  PROCESSES  IN  DEVELOPMENT  PHASES  OF 
PROGRAMS  AND  THE  PRODUCT  IMPROVEMENT  OF  ITEMS 


Reliability  growth  in  a population  of  items  occurs  as  a result  of  the 
correction  of  deficiencies  through  redesign  and  retrofit.  In  development 
programs,  this  is  commonly  termed  the  "Test  - Fix,  Test  - Fix"  process; 
for  systems  in  operational  service  we  recognize  this  process  as  one  aspect 
of  the  product  improvement  cycle.  We  must  recognize  a fact  of  life  in  the 
design  and  development  of  todays  complex,  high  performance  system. 

Although  the  developer,  either  Army  or  Contractor,  will  have  done  his 
level  best  to  "design" nigh  reliability  into  the  device  in  a sincere  aim 
to  meet  the  operational  reliability  goals, the  economic  and  timing  constraints 
as  well  as  the  usual  departure  of  new  designs  from  earlier  practice 
resulting  from  the  technical  challenges  , there  will  appear  a series  of 
unexpected  failure  modes  once  the  device  is  put  on  test,  first  in  the 
laboratory,  and  later  in  the  field.  True,  a number  of  Product  Assurance 
practices  such  as  failure  mode  and  effects  analyses,  fault  tree  analyses, 
and  periodic  in-depth  design  reviews  have  as  their  aim  the  minimization 
of  design  oversights,  there  will  still  exist  a distribution  of  failure 
modes  with  varying  failure  rates  which  are  waiting  to  be  discovered  and 
corrected  by  design  ch.anges.  We  can  visualize  the  reliability  growth 
process  then  as  the  "erosion"  of  the  failure  rates  distribution  as  test 
hours  and  later  field  hours  are  accumulated.  We  can  visualize  this  process 
by  depicting  such  a hypothetical  failure  rate  distribution  as  shown  in 
Figure  1. 
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Failure  Rate  - Failures/10^  Hours 
Figure  1 


Therefore,  for  ar.y  cue  specific  configuration,  the  reliability  should 
net  be  expected  to  grew  during  even  an  extensive  series  of  tests  unless, 
of  course,  items  subsequent  to  the  appearance  of  particular  failures  have 
been  retrofitted  vith  the  particular  corrective  features. 

The  question  on  how  quality  control  caused  problems  should  be  treated 
has  come  up.  Quality-caused  failures  should  be  considered  as  evidences 
that  the  device  in  question  is  not  representative  of  the  design.  Thus 
manufacturing  in  fact  has  not  produced  a product  that  meets  the  design 
specification.  A typical  example  might  be  the  poor  apparent  reliability 
cf  tne  DRAGON  missile.  In  this  case,  an  aggressive  tightening  of  quality 
control  with  some  furtcer  organizational  changes  resulted  in  a product 
which  finally  met  the  design  requirement  and  performed  to  expectations. 

A very  important  requirement  for  the  effective  tracking  of  reliability 

growth  is  a clear  set  of  definitions  of  what  constitutes  mission  success 

and  what  criteria  of  measurement  are  to  be  used.  In  essentially  all  of 

my  discussions  with  Project  Managers,  this  item  of  concern  was  raised  as 

a major  factor.  Failure  definitions  were  seldom  clear  and  unambiguous, 

often  the  key  performance  factors  were  not  even  included.  Thus, for  an 
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aircraft  like  UTTAS,  MTBF  to  complete  a mission  is  simply  not  a sufficient 
evaluator  of  the  state  of  the  system.  Should  failure  of  an  indicator 
light  bulb  be  treated  as  having  the  same  gravity  as  failure  of  an  engine 
bearing? 

Thus,  additional  measures  such  as  the  Important  removal  rate,  main- 
tenance Indicators  such  as  maintenance  man-hours  per  flight  hour,  and 
possibly  availability,  should  be  developed  and  tracked.  Such  additional 
measures  would  give  the  project  manager  a far  better  means  of  evaluating 
the  true  effectiveness  and  rate  of  growth  of  his  product. 

The  rate  of  reliability  growth  af  any  one  time  in  a program  depends 
on  several  factors.  I have  already  mentioned  the  responsiveness  of  the 
program  in  fixing  problems  and  retrofitting  items.  Another  factor  is 
the  test  severity.  If  the  severity  is  high,  new  failure  modes  will  appear 
at  a higher  rate  per  test  hour  than  would  be  the  case  for  test  conditions 
that  are  less  severe  than  for  example  normal  duty.  Consideration  should, 
therefore,  be  given  tr  the  establishment  of  test  hour  multipliers  to  account 
for  this  well  known  factor  and  perttit  a realistic  assessment  of  development 
program  reliability  prtgress. 

We  must  rscognita  that  an  effective  reliability  growth  management 
program  requires  an  entelient  failure  and  degradation  reporting  effort 
as  well  as  a timely  analysis  of  all  events  and  resulting  maintenance  or 
corrective  actions.  Much  experience  Is  being  gained  in  this  area  and 
these  lessens  must  be  transmitted  throughout  DARCOM, 

In  conclusion,  1 get  the  clear  message  that  reliability  growth 

management  is  becoming  a very  useful  approach  to  accelerate  the  development 

progress  in  todays  complex  weapons  systems.  The  philosophy  and  techniques 

are  rather  new  and  are  understood  to  varying  degrees  in  different  projects. 
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Reliability  growth  management  is  rapidly  becoming  one  of  the  key  drivers 
in  project  management,  dictating  many  major  decisions  on  the  allocation 
of  resources.  As  experience  is  gained  and  greater  insight  into  the 
reliability  growth  processes  is  obtained,  refinements  in  the  technical 
aspects  must  be  made  to  further  facilitate  the  decision  processes.  Thus, 
resources,  on  a continuing  basis,  must  be  provided  to  assure  the  timely 
and  effective  development  of  the  techniques  and  the  disemination  to  the 
projects  via  symposia  and  direct  input  from  the  technical  focal  point; 
namely,  AMSAA. 
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FINDINGS 


1)  RELIABILITY  GROWTH  TECHNOLOGY  HAS  REACHED  A LEVEL  OF  DEVELOPMENT 
WHICH  IS  STATISTICALLY  SOUND  AND  REASONABLY  ADEQUATE  TO  THE  TASKS.. 

Discussion 

The  so-called  AMSAA  model  of  reliability  growth  appears  to  be  technically 
sound.  However,  a continuing  effort  is  needed  to  further  develop  confidence 
bounds  for,  particularly,  truncated  tests.  Procedures  should  be  developed 
for  small  sample  sizes  so  that  risks  in  data  Interpretation  can  be  Identified. 
Sensitivity  of  the  goodness  to  fit  test  used  should  be  established  so  that 
deviations  of  measured  data  from  the  model  can  be  Interpreted  in  terms  of 
risks.  The  question  of  how  to  handle  deviations  from  exponential  reliability 
model  such  as  distinct  wearout  patterns  with  increasing  part  age  should  be 
addressed. 

In  summary,  development  work  on  the  mathematical  - statistical  aspects 
of  the  growth  model  must  continue  to  Incorporate  experience  which  is 
currently  gained.  Ah5.-Ji  is  at  present  not  staffed  to  perform  this  Important 
function.  Dr.  Larry  Crow,  one  of  the  key  contributors  to  reliability  growth 
modeling  is  not  spending  any  significant  amount  of  time  on  the  further 
development  of  the  technique.  Also,  AMSAA  must  continue  to  act  as  teacher, 
and  to  some  degree,  coordinator,  between  particularly  the  test  agencies 
such  as  OTEA  and  TECOM. 

Recoounendat  ions 

a)  An  identified, continuing  effort  should  be  carried  out  at  AMSAA 
to  further  develop  and  verify  the  AMSAA  Reliability  Growth  Model* 

This  should  be  a continuing,  planned  program  with  at  least  one 
full  time  bright  young  engineer  working  under  the  direction  of 


Dr.  Crow. 
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b)  Charge  the  AMSAA  function  with  responsibility  to  assure  a co- 
ordinated application  of  reliability  growth  methodology  in 
various  test  operations. 

c)  Consider  contracting  with  a University  or  one  of  the  DARCOM 
contractors  who  are  knowledgeable  in  realiability  techniques. 

to  contribute  to  the  development  of  reliability  growth  techniques. 
Possibilities  are  The  University  of  California  at  Berkely;  the 
Rand  Corporation;  the  United  Technology  Center  at  Sunnyvale, 
California;  Texas  A & M;  Penn  Stated  Massachusetts  Institute 
of  Technology,  Nuclear  Engineering  Department;  The  University 
of  Arizona  at  Tucson. 

2)  THE  LEVEL  OF  UirDSRST.lliDING  OF  RELIABILITY  GROLTH  CONCEPTS  AND 

TECHNIQUES  V.ARIES  FROM  COMMANT)  TO  C0>2iAND  A1!D  IN  PROJECT  MANAGER 
ORGANIZATIONS.  IN  SO.ME  CASES,  THE  UNDERSTANDING  IS  SUPERFICIAL 
AND  HAS  RESIIIZD  IN  THE  SETTING  OF  UNREALISTIC  RELIABILITY 
OBJECTIVES. 

Discussion 

Reliability  growth  tanagenent  techniques  are  relatively  new  and 
rapresent  inovative  tnlr.King  in  the  management  of  development  and  product 
improvement  prcgracs.  I have  detected  some  rather  distressing  mis- 
conceptions about  the  basic  reliability  growth  processes  among  high  level 
managerial  and  working  level  technical  personnel.  On  the  ocher  hand,  I have 
also  met  with  a profound  appreciation  of  the  opportunities  of  the  methods 
laced  with  a realistic  bit  of  scepticism  on  the  part  of  several  Project 
Managers.  The  power  of  the  method,  as  well  as  its  limitations,  must  be 
clearly  understood  by  the  user. 

I 

I 

I 

t 

f 


For  example,  in  a particular  project,  surprise  was  expressed  that 
no  reliability  fjrowth  was  experienced  in  one  particular  series  of  extensive 
tests  involving  several  items  of  equipment  with  no  corrective  action 
taken  during  the  test.  In  another  case,  the  reliability  growth  projections 
to  introduction  into  operational  service  ("the  project  budgeted  growth 
curve")  toward  meeting  the  Material  Need  document  requirements  showed 
a completely  unrealistic  slope  of  2.4.  Interestingly,  the  reliability 
growth  curve  slope  up  through  the  DT/OT  II  was  a quite  realistic  0.752. 

The  only  conclusion  which  the  Project  Manager  should  have  drawn  was  that 
the  MN  requirements  sinply  could  not  be  achieved  with  the  normal  develop- 
ment processes  and  that  a step  increase  in  reliability  must  be  budgeted 
for  by  planning  on  a major  review  of  all  failures,  indications  of  design 
weaknesses  and  the  required  design  improvements  and  verification  by  an 
extensive  test  program.  Based  on  the  Initial  low  levels  of  reliability, 
it  was  apparent  that  the  MN  requirements  could  simply  not  be  net  if  the 
assumption  of  normal  reliability  growth  rates  could  be  verified  as  being 
applicable. 

The  process  of  ecucating  the  DARCOM  community  in  the  application  of 
reliability  growth  management  and  tracking  techniques  must  continue, 
possibly  at  an  accelarated  pace.  The  two  symposia  appear  to  have  been 
well  conceived  and  well  executed.  These  should  be  periodically  repeated. 

In  addition,  however,  workshops  should  be  held  at  the  commands  to  educate 
the  technical  personnel  in  the  fundamentals  and  the  application  of  the 
techniques.  Of  particular  importance  is  the  training  of  project  product 
assurance  personnel  so  that  the  decision-makers  are  properly  Informed. 
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Recommendations 


a)  AMSAA  should  be  stalled  and  lunded  to  prepare  and  hold  periodic 
workshops  at  the  various  commands  ol  DARCOM  and  the  project 
managers  0111 ces. 

b)  AMSAA  should  hav’e  the  opportunity  to  periodically  assess  the 
technical  soundness  ol  the  reliability  growth  techniques 
application  at  the  various  project  ollices. 

c)  The  test  operators,  in  particular  TECOM  and  OTEA,  must  be 
schooled  in  the  statistical  techniques  needed  to  properly  track 
reliability  growth. 

d)  Up  to  date  sranuals  should  be  prepared  which  are  strictly 
application-oriented  lor  use  ol  DARCOM  personnel.  1 under- 
stand that  Dr.  Crow  has  already  begun  this  work  but  needs 
additional  time  to  complete  the  task. 

3)  FAILURE  DSEIiriTIOMS  /JIE  NOT  ALWAYS  CLEAR  IN  CONTRACTS  WITH  CON- 
TRACTORS AND  H.V1E  RESULTED  IN  MJCK  MISUSE  OF  RELIABILITY  DATA  AND 
IN  F.ACT  H.WE  DETRACTED  FROM  THE  CREDIBILITY  OF  THE  APPROACH  AS 

l'ni;eczssary  .-.rgtments  ensured. 

Discussion 

The  need  for  clear  failure  definition  and  mission  definitions  must 
be  stressed.  These  definitions  must  be  developed  in  the  negotiation  phases 
of  contracts  and  included  in  contractual  documents.  Scoring  conferences 
must  continue  to  be  used  by  all  activities  tracking  reliability  growth. 
Thus,  the  Letter  of  Agreement  (LOA)  and  the  subsequent  ROC  documents  must 
include  the  appropriate  measures  of  reliability  and  maintainability. 
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The  reliability  of  an  end  item  should  be  expressed  In  truly  operational 
^ terms.  Thus,  mean  time  between  failure  may  not  be  a sufficient  indicator. 

For  aircraft  engines,  for  example,  mean  time  between  power  loss,  mean  time  - 
between  in-flight  shutdown,  removal  rate,  and  maintenance  man  hours  per 
flight  hour  are  used  together  as  criteria.  In  some  situations,  availability 
may,  in  fact,  be  an  appropriate  measure.  TRADOC  should  play  an  important 
role  in  this  process. 

In  my  discussions  with  the  Project  Managers  I found  almost  universal 
concern  about  the  need  for  precise  and  definitive  failure  definitions.  It 
would,  therefore,  be  advisable  to  develop  a series  of  guidelines  on  the 
type  of  failure  definitions  and  their  format  for  various  type  weapons  and 
.■r-  systems.  The  P.M.'s  feel  that  the  failure  definitions  need  to  be  refined 

as  experience  is  gained  with  the  tracking  of  reliability  growth.  There  is 
[ also  concern  about  differing  interpretations  of  the  failure  definitions 

by  first  the  developer  and  then  the  user.  Their  outlook  differs  as  a result 
of  their  different  basic  orientations. 

The  question  of  purging  the  data  of  failures  that  are  in  the  process  of 
correction  came  up  several  tines.  It  should  be  made  clear  that  the  AMSAA 
vor  Duane)  model  dees  net  permit  purging  of  data  as  the  result  would  be 
injustly  biased.  Similarly,  test  data  from  high  stress  environmental  tests 
should  be  carefully  screened.  If  the  test  condition  is  truly  not  representative 
of  operational  conditions,  then  such  data  may  be  excluded  unless,  in  the 
opinion  of  responsible  technical  personnel,  a potential  problem  has  been 
uncovered . 

Recommendations : 

a)  I ’.itiate  a study  by  AMSAA,  jointly  with  TRADOC  and  several  Commands, 
to  develop  a set  of  guidelines  for  the  definition  of  failure 
I criteria  and  reliability  measures. 
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b)  Include  carelully  developed  failure  definitions  and  reliability 


criteria  in  LOA  and  KOC  documents. 

c)  Develop  guidelines  for  the  incorporation  of  reliability  measures 
into  contract  documents. 

A)  AS  A RESULT  OF  AFiARC  AND  PREVIOUS  REORGANIZATIONS,  RELIABILITY 
TALENT  HAS  BEEN  DIFFUSED  TO  THE  DETRIMENT  OF  THE  EFFECTIVE  IM- 
PLEMENTATION 0?  RELIABILITY  GROWTH  TECHNOLOGY. 

Discussion 

Concern  has  been  expressed  at  both  AMSAA  and  the  Quality  Assurance 
Directorate  about  the  deleterious  effect  of  the  reorganizations  on  the 
staff  capabilities  of  these  two  key  organizations.  It  is,  of  course, 
inperatlve  that  a capable  cadre  be  maintained  to  perform  the  necessary 
support  functions.  I am.  informed  that  several  very  capable  young  engineers 
and  mat  lematicians  were  relocated. 

Recommendat ions 

a)  Have  AMS.^A  and  Quality  .Assurance  present  their  cases  to  DARCOM 
management  to  assure  that  the  scope  of  the  problem  is  defined 
so  that  corrective  action  can  be  taken. 

5)  RELIA3ILIT”  GROWTH  MANAGEMENT  TECHNIQUES  ARE  APPLIED  OVER  THE 
BRO.AD  FRD.VT  TR  D.ARCOM  ACTIVITIES  WITH  CONSIDERABLE  PAYOFF  IN 
increased  TATE  0?  RELIABILITY  IMPROVEMENT  IN  DEVELOPMENT.  YET 
FTlCH  IMPRO'TMENT  SHOULD  STILL  BE  ENCOURAGED  TO  DEVELOP  FULL 
POTENTIAL  OF  THE  METHODS. 

Discussion 

The  reaction  to  the  introduction  of  the  Reliability  Growth  Management 


techniques  into  the  project  manager's  portfolio  of  managerial  tools  varies 
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from  tremendous  enthusiasm  to  "its  been  foisted  on  us  and  we  can  live  with 


it."  The  enthusiastic  supporters  have  stated  that  the  reliability  growth 
teclinique  has  been  of  the  key  driving  factors  in  the  program  and  has  most 
certainly  been  a major  contributor  to  rapid  reliability  Improvement.  There' 
has  been  some  difficulty  in  early  implementation  in  a program  because 
of  the  dearth  of  a good  data  base  but  nonetheless  it  is  generally  agreed 
that  reliability  growth  monitoring  is  becoming  a useful  approach.  Some 
of  the  other  difficulties  have  already  been  discussed  in  the  previous  dis- 
cussions in  this  report  and  the  previous  findings. 

There  is  simply  no  doubt  that  reliability  growth  management  has  con- 
tributed to  reductions  in  program  costs  as  expressed  by  UTTAS,  Stinger, 
Shillelagh,  SAM-D,  TACFIF.S,  DRAGON,  MICV  and  M60  programs  to  cite  a few 
specific  instances. 

I have  found,  in  the  negative  reactions,  that  they  are  based  on  an 
incomplete  and,  at  times,  erroneous  understanding  of  the  fundamental  processes 
and  even  intents  of  the  approach. 

The  stage  has  been  set  Co  make  broader  use  of  the  technique  as  a 
management  tool  for  better  risk  assessment  and  resource  allocation  through 
further  education  and,  in  particular,  the  development  of  development  program 
simulation  techniques.  Such  stochastic  simulations  will  permit  the  more 
effective  allocation  of  resources  to  maximize  reliability  and  availability 
achievement . 

Recommendat ions 

a)  Continue  to  encourage  the  use  of  reliability  growth  techniques  by 
incorporation  into  Army  Regulations,  contractual  documents, and 
specifications. 

b)  Continue  the  development  of  the  technique  on  a continuing  basis 
by  collecting  experience  on  current  programs  and  through  use  of 
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advanced  statistical  methods. 

c)  Provide  for  effective  dissemination  of  successful  practices  to 
new  programs. 
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